Use page tracking for snapshot and restore #683

simongdavies · 2025-07-01T20:40:44Z

This pull request introduces snapshotting and restoring of sandbox state using dirty page tracking rather than copying the entire memory state each time, in many/most scenarios where sandboxes have larger than default amounts of memory this results in better perforamnce and reduced memory usage.

However, due to inefficiencies in the way dirty page tracking works on mshv and the fact that the implementation of tracking is not done for Windows this is not universally true.

Included below are screenshots showing the difference in time taken for some benchmarks with and without dirty page tracking enabled, along with some explanations of the differences seen.

The changes comprise:

Host Shared Memory Dirty Page tracking

Added a custom signal handler for SIGSEGV to support dirty page tracking for host memory mapped into a VM. Updated documentation to reflect this change and added debugging instructions for handling SIGSEGV in GDB and LLDB.

VM Dirty page tracking

Enabled dirty page tracking for mshv and KVM drivers to track changes made to memory in the guest

Snapshot Management

Added a new snapshot manager module to create, manage and restore memory snapshots.

Benchmarking Enhancements

Refactored the guest_call_benchmark_large_param function to support benchmarks for multiple parameter sizes and added a new sandbox_heap_size_benchmark function to measure sandbox creation performance with varying heap sizes.
Introduced guest_call_heap_size_benchmark to evaluate guest function call performance with different heap sizes.

Performance Changes

Expand for older details

KVM Before

KVM After

KVM shows massive positive changes in performance when creating large sandboxes and calling functions in sandboxes with large memory configurations , although not measured the amount of memory consumed should have reduced considerably. In the scenario where large parameters are passed to the sandbox there is a degradation of performance, this has not been investigated yet, it may be due to the page based mechanism for saving/restoring data being more expensive than copying and restoring all the data. Other work to make large parameter passing more efficient will likely have a large positive impact here as well.

There is some regression in performance for small/default sandbox sizes which is more than likely caused by the overhead of tracking and building/restoring page based snapshots for small memory vs. copying and restoring the entire memory.

mshv2 Before

mshv2 After

mshv3 Before

mshv3 After

Both mshv2 and mshv3 show similar patterns to KVM in that the larger sandbox sizes show large improvements, the default/small sandboxes show regressions and the large parameter sizes show regressions, possibly for the same reason that the KVM one does , again no investigation done here yet.

The biggest difference between KVM and mshv is for two reasons, first when enabling dirty page tracking on mshv the first call to get dirty pages results in a returned bitmap showing all pages dirty, since this would cause us to snapshot all memory as a baseline after enabling dirty page tracking this PR gets the dirty pages immediately and then discards the result. This approach means that we have to make 2 calls to get dirty pages when we really only need one. The impact of this is quite large, especially with larger memory configurations since the call to get dirty pages seems to be O(n) where n is the number of pages in the memory configuration (regardless of if the pages are dirty or not), for a 950mb VM I observed ~1.4ms response for this call, however, this approach with larger sandboxes is still much quicker than copying all the memory. Fixing these issues in mshv will probably bring the performance much closer to KVM.

Windows 2025 Before

Windows 2025 After

Windows is largely either the same performance or has regressed, this is because the Windows implementation has not been done yet, at the moment each time dirt pages are requested Windows reports that all pages are dirty and the snapshots/restores are done on that basis, there is some overhead of this approach especially when restoring

ludfjig

Looks mostly good to me. The signal handling of SIGSEGV makes me a little nervous, and there seems to be an assumption that Vec<MemoryRegions> are always consecutive in memory, which i think we might break in the future. I also think a refactor of our memory management code would be nice 😅 .

I also remember I had a bunch of tests for evolve/devolve in my previous dirty-pages PR that maybe could be useful to test logic. Those tests were the ones that allowed me to catch the bug in mshv's implementation :P

Also, these regression seems pretty big (compared to main branch)

src/hyperlight_host/src/mem/shared_mem.rs

src/hyperlight_host/src/mem/linux_dirty_page_tracker.rs

ludfjig · 2025-07-03T18:54:03Z

src/hyperlight_host/src/hypervisor/kvm.rs

+            for page_idx in bit_index_iterator(&bitmap) {
+                page_indices.push(current_page + page_idx);
+            }
+            current_page += num_pages;


I think we are making an assumption here that all memory regions are consecutive with no gaps. I think Vec<MemoryRegion> has this invariant, but not sure if documented anywhere

src/hyperlight_host/src/mem/mgr.rs

ludfjig · 2025-07-03T19:02:58Z

src/hyperlight_host/src/mem/shared_memory_snapshot_manager.rs

+            shared_mem.with_exclusivity(|e| e.copy_to_slice(&mut buffer, 0))??;
+        } else {
+            // Sort pages for deterministic ordering and to enable consecutive page optimization
+            dirty_pages.sort_unstable();


I believe this sort is redundant

ludfjig · 2025-07-03T19:15:01Z

src/hyperlight_host/src/mem/shared_memory_snapshot_manager.rs

+
+        // Collect dirty pages and sort them for consecutive page optimization
+        let mut dirty_pages: Vec<usize> = bit_index_iterator(dirty_bitmap).collect();
+        dirty_pages.sort_unstable();


I believe this sort is redundant

src/hyperlight_host/src/mem/mgr.rs

ludfjig · 2025-07-03T19:25:24Z

src/hyperlight_host/src/mem/shared_memory_snapshot_manager.rs

+    pub(super) fn create_new_snapshot<S: SharedMemory>(
+        &mut self,
+        shared_mem: &mut S,
+        dirty_page_map: Option<&Vec<u64>>,


I think(?) this option can also be removed

src/hyperlight_host/src/mem/shared_memory_snapshot_manager.rs

ludfjig · 2025-07-03T19:28:59Z

src/hyperlight_host/src/hypervisor/hyperv_windows.rs

@@ -606,6 +607,12 @@ impl Hypervisor for HypervWindowsDriver {
        Ok(())
    }

+    fn get_and_clear_dirty_pages(&mut self) -> Result<Vec<u64>> {
+        // For now we just mark all pages dirty which is the equivalent of taking a full snapshot
+        let total_size = self.mem_regions.iter().map(|r| r.guest_region.len()).sum();


Can we use https://learn.microsoft.com/en-us/virtualization/api/hypervisor-platform/funcs/whvquerygparangedirtybitmap ?

…ctions. Signed-off-by: Ludvig Liljenberg <[email protected]>

…nto shared memory. Signed-off-by: Ludvig Liljenberg <[email protected]>

Signed-off-by: Ludvig Liljenberg <[email protected]>

…Uninstall DirtyPageTracker immediately before setting up VM and mapping memory into VM. Use dirty pages to create snapshots with new snapshotmanager. Signed-off-by: Ludvig Liljenberg <[email protected]> Move setting up page tables out of set_up_hypervisor_partition Signed-off-by: Ludvig Liljenberg <[email protected]>

…parameters Signed-off-by: Ludvig Liljenberg <[email protected]>

Signed-off-by: Ludvig Liljenberg <[email protected]>

simongdavies added the kind/enhancement For PRs adding features, improving functionality, docs, tests, etc. label Jul 1, 2025

simongdavies linked an issue Jul 1, 2025 that may be closed by this pull request

Improve performance when creating "larger" Sanboxes #642

Open

simongdavies force-pushed the update-snapshot-and-restore branch 5 times, most recently from 7e85120 to 014eab3 Compare July 3, 2025 15:48

This was referenced Jul 3, 2025

Implement Dirty Page Tracking on Windows #688

Open

Create repro for mshv get dirty page issues #689

Open

Investigate why page tracking changes make calling functions with large parameters slower than previous implementation #690

Open

simongdavies changed the title ~~WIP: Use page tracking for snapshot and restore~~ Use page tracking for snapshot and restore Jul 3, 2025

simongdavies force-pushed the update-snapshot-and-restore branch from 014eab3 to 56fd983 Compare July 3, 2025 16:43

ludfjig reviewed Jul 3, 2025

View reviewed changes

ludfjig force-pushed the update-snapshot-and-restore branch 5 times, most recently from 1231cc6 to 9d4d90a Compare July 8, 2025 21:08

Add API for getting VM's dirty pages, and add some bitmap utility fun…

ea6be2e

…ctions. Signed-off-by: Ludvig Liljenberg <[email protected]>

ludfjig force-pushed the update-snapshot-and-restore branch from 9d4d90a to 98e4e67 Compare July 9, 2025 17:57

ludfjig added 2 commits July 9, 2025 12:18

Add signal-based dirty page tracker for tracking host-memory-writes i…

96754b8

…nto shared memory. Signed-off-by: Ludvig Liljenberg <[email protected]>

Add new SnapshotManager and Snapshot structs.

5d71e2a

Signed-off-by: Ludvig Liljenberg <[email protected]>

ludfjig force-pushed the update-snapshot-and-restore branch from 98e4e67 to 5430060 Compare July 9, 2025 19:18

ludfjig added 5 commits July 9, 2025 22:02

Add new benchmarks for varying sizes of sandboxes and guest function …

6de0d2f

…parameters Signed-off-by: Ludvig Liljenberg <[email protected]>

Update docs about signal handlers

39a5d3e

Signed-off-by: Ludvig Liljenberg <[email protected]>

Add 1 integration test

cdf8b02

Signed-off-by: Ludvig Liljenberg <[email protected]>

Add debugging function for visualizing dirty pages in a specific layout

eb9f57a

Signed-off-by: Ludvig Liljenberg <[email protected]>

ludfjig force-pushed the update-snapshot-and-restore branch from 5430060 to eb9f57a Compare July 10, 2025 05:02

[temp-commit] fail-fast: false

d772aef

Signed-off-by: Ludvig Liljenberg <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Use page tracking for snapshot and restore #683

Use page tracking for snapshot and restore #683

simongdavies commented Jul 1, 2025 •

edited by ludfjig

Loading

Uh oh!

ludfjig left a comment •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Uh oh!

ludfjig Jul 3, 2025

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Uh oh!

Uh oh!

Use page tracking for snapshot and restore #683

Are you sure you want to change the base?

Use page tracking for snapshot and restore #683

Conversation

simongdavies commented Jul 1, 2025 • edited by ludfjig Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Host Shared Memory Dirty Page tracking

VM Dirty page tracking

Snapshot Management

Benchmarking Enhancements

Performance Changes

KVM Before

KVM After

mshv2 Before

mshv2 After

mshv3 Before

mshv3 After

Windows 2025 Before

Windows 2025 After

Uh oh!

ludfjig left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

ludfjig Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ludfjig Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

simongdavies commented Jul 1, 2025 •

edited by ludfjig

Loading

ludfjig left a comment •

edited

Loading